Echo State Network Research Articles

Impact of time-history terms on reservoir dynamics and prediction accuracy in echo state networks.

The echo state network (ESN) is an excellent machine learning model for processing time-series data. This model, utilising the response of a recurrent neural network, called a reservoir, to input signals, achieves high training efficiency. Introducing time-history terms into the neuron model of the reservoir is known to improve the time-series prediction performance of ESN, yet the reasons for this improvement have not been quantitatively explained in terms of reservoir dynamics characteristics. Therefore, we hypothesised that the performance enhancement brought about by time-history terms could be explained by delay capacity, a recently proposed metric for assessing the memory performance of reservoirs. To test this hypothesis, we conducted comparative experiments using ESN models with time-history terms, namely leaky integrator ESNs (LI-ESN) and chaotic echo state networks (ChESN). The results suggest that compared with ESNs without time-history terms, the reservoir dynamics of LI-ESN and ChESN can maintain diversity and stability while possessing higher delay capacity, leading to their superior performance. Explaining ESN performance through dynamical metrics are crucial for evaluating the numerous ESN architectures recently proposed from a general perspective and for the development of more sophisticated architectures, and this study contributes to such efforts.

Finite-time event-triggered output-feedback adaptive decentralized echo-state network fault-tolerant control for interconnected pure-feedback nonlinear systems with input saturation and external disturbances: A fuzzy control-error approach

In this paper, we introduce a decentralized Event-Triggered fault-tolerant echo-state network (ESN) direct adaptive controller for uncertain interconnected nonlinear systems in pure-feedback form. The proposed controller addresses input saturation, actuator faults, external disturbances, and unavailable states for measurement. Unlike the existing works in the literature that adapts the ESN weights using the tracking error, our method employs the control error that is estimated using a fuzzy inference system, to derive the adaptation laws. The complexity explosion typically seen with recursive back-stepping and Dynamic Surface Control (DSC) designs is completely avoided. Our control scheme addresses four types of state-dependent actuator faults: bias, drift, loss of accuracy, as well as loss of effectiveness. The ESNs approximate unknown ideal control laws, while robust terms are added to enhance the stability of the closed-loop system. Stability analysis ensures that tracking errors converge, in finite time, to a small compact set near the origin due to the strictly positive real (SPR) property. Simulation results of a quadrotor UAV and a mathematical system are presented. A comparative analysis is performed with another approach to emphasize the effectiveness of the proposed method.

Neural Networks with Transfer Learning and Frequency Decomposition for Wind Speed Prediction with Missing Data

This paper presents a novel data-driven approach for enhancing time series forecasting accuracy when faced with missing data. Our proposed method integrates an Echo State Network (ESN) with ARIMA (Autoregressive Integrated Moving Average) modeling, frequency decomposition, and online transfer learning. This combination specifically addresses the challenges missing data introduce in time series prediction. By using the strengths of each technique, our framework offers a robust solution for handling missing data and achieving superior forecasting accuracy in real-world applications. We demonstrate the effectiveness of the proposed model through a wind speed prediction case study. Compared to the existing methods, our approach achieves significant improvement in prediction accuracy, paving the way for more reliable decisionmaking in wind energy operations and management.

A new two-stage decomposition and integrated hybrid model for short-term wind speed prediction

Accurate wind speed prediction is of essential importance for the stability and safe operation of power systems. Given the complexity of wind speed sequence, this paper proposed a new two-stage decomposition and integrated hybrid model to improve the accuracy of wind speed prediction. A two-stage decomposition method combining robust local mean decomposition (RLMD), sample entropy (SE) and variational modal decomposition (VMD) was used to decompose the wind speed signal in the data preprocessing stage. Firstly, the wind speed signal was decomposed into various components by RLMD, and the complexity of each component was calculated using the SE to classify them into random, detail component and trend component. Then, a secondary decomposition of the random component with the highest SE was performed using the VMD. In the prediction stage, two different prediction models were used for prediction depending on the smoothness of each component. Stochastic configuration networks (SCN) was used to predict the detail and trend components with relatively smoothness. Echo state network (ESN) was used to predict the components of the secondary decomposition. Finally, the actual wind speed data were compared by different prediction models, which illustrated that the prediction method proposed in this paper had good prediction accuracy and generalizability.

An echo state network with interacting reservoirs for modeling and analysis of nonlinear systems

An echo state network with interacting reservoirs for modeling and analysis of nonlinear systems

Multi-state delayed echo state network with empirical wavelet transform for time series prediction

Multi-state delayed echo state network with empirical wavelet transform for time series prediction

Investigating forced transient chaos in monsoon using Echo State Networks

Investigating forced transient chaos in monsoon using Echo State Networks

Echo state network and classical statistical techniques for time series forecasting: A review

Forecasting is an extensive field of study, which tries to avoid injuries, diseases, and damages but also can help in energy production, finance investments, etc. Two mathematics modeling techniques have obtained promising results: the ones based on Machine Learning (Echo State Network) and based on Statistical techniques (ARIMA/GARCH). To take advantage of both techniques, we aimed to perform a systematic literature review of Echo State Network and classical Statistical techniques for forecasting Time Series. We conducted the searches on the databases ACM, IEEE Xplore, Scopus, and Web of Science and, after, did a bibliometric and a content qualitative analysis of the selected articles. We present the techniques and sources of the data set used, the most used keywords in the articles, analyze the reservoir computing/echo state network and statistical techniques, and comment on each article selected. From the analysis of this review, it is possible to infer that it is still an area to be studied more deeply and that the academy, even if timidly, never stopped using the echo state network for time series regression in general and financial series.

A novel broad learning system integrated with restricted Boltzmann machine and echo state network for time series forecasting

Time series data prediction is crucial in system control, social management, and economic production. For the complex features of time series data and the massive amount of arithmetic in deep learning, a novel network model is proposed based on the broad learning architecture to handle time series prediction tasks. The model utilizes the restricted Boltzmann machine (RBM) in the mapping layer to learn feature information from the input data. Simultaneously, it employs the echo state network (ESN) as the fundamental unit in the enhancement layer to fit the learned feature information from the mapping layer. The proposed model's predictive performance has been validated on air quality index, PM2.5, and electric power datasets. Compared with the benchmark model, it has the advantage of training speed and can reduce prediction error by up to 36%, proving its effectiveness in time series prediction tasks.

Newman-Watts-Strogatz topology in deep echo state networks for speech emotion recognition

Cortical neural connectivity exhibits a small-world (SW) network topology. However, its role in neural information processing remains unclear. In this study, we investigate the learning performance of a Deep Echo State Network (DeepESN) that incorporates the Newman-Watts-Strogatz (NWS) graph topology as a reservoir. The short path length of the SW topology allows input information to reach output nodes faster, and the clustered arrangement of this topology can expand the range of the echo state property. Additionally, in this paper, we seek to identify the best set of these key factors, such as the spectral radius of the reservoir’s weight matrix and the hyperparameters neighbors' and probabilities. We evaluate the NWS-DeepESNs on four Speech Emotion Recognition (SER) tasks, which are RAVDESS, Emo-db, SAVEE, and TESS databases. By examining the NWS-DeepESN’s performance on SER tasks, we aim to determine the role of the NWS graph topology in enhancing neural information processing. The obtained results exhibited significant improvements in recognition accuracies (weighted and unweighted) with (76.45%, 75.79%), (87.89%, 87.14%), (99%, 98.62%), and (63%, 62.24%), respectively, for RAVDESS, Emo-db, SAVEE, and TESS databases. The proposed approach improves emotion recognition capabilities and yields more accurate results compared to existing state-of-the-art approaches.

An ensemble of artificial neural network models to forecast hourly energy demand

AbstractWe propose an ensemble artificial neural network (EANN) methodology for predicting the day ahead energy demand of a district heating operator (DHO). Specifically, at the end of one day, we forecast the energy demand for each of the 24 h of the next day. Our methodology combines three artificial neural network (ANN) models, each capturing a different aspect of the predicted time series. In particular, the outcomes of the three ANN models are combined into a single forecast. This is done using a sequential ordered optimization procedure that establishes the weights of three models in the final output. We validate our EANN methodology using data obtained from a A2A, which is one of the major DHOs in Italy. The data pertains to a major metropolitan area in Northern Italy. We compared the performance of our EANN with the method currently used by the DHO, which is based on multiple linear regression requiring expert intervention. Furthermore, we compared our EANN with the state-of-the-art seasonal autoregressive integrated moving average and Echo State Network models. The results show that our EANN achieves better performance than the other three methods, both in terms of mean absolute percentage error (MAPE) and maximum absolute percentage error. Moreover, we demonstrate that the EANN produces good quality results for longer forecasting horizons. Finally, we note that the EANN is characterised by simplicity, as it requires little tuning of a handful of parameters. This simplicity facilitates its replicability in other cases.

Improving Intrusion Detection using Satin Bowerbird Optimization with Deep Learning Model for IIoT Environment

Intrusion Detection in the Industrial Internet of Things (IIoT) concentrations on the security and safety of critical structures and industrial developments. IIoT extends IoT principles to industrial environments, but linked sensors and devices can be deployed for monitoring, automation, and control of manufacturing, energy, and other critical systems. Intrusion detection systems (IDS) in IoT drive to monitor network traffic, device behavior, and system anomalies for detecting and responding to security breaches. These IDS solutions exploit a range of systems comprising signature-based detection, anomaly detection, machine learning (ML), and behavioral analysis, for identifying suspicious actions like device tampering, unauthorized access, data exfiltration, and denial-of-service (DoS) attacks. This study presents an Improving Intrusion Detection using Satin Bowerbird Optimization with Deep Learning (IID-SBODL) model for IIoT Environment. The IID-SBODL technique initially preprocesses the input data for compatibility. Next, the IID-SBODL technique applies Echo State Network (ESN) model for effectual recognition and classification of the intrusions. Finally, the SBO algorithm optimizes the configuration of the ESN, boosting its capability for precise identification of anomalies and significant security breaches within IIoT networks. By widespread simulation evaluation, the experimental results pointed out that the IID-SBODL technique reaches maximum detection rate and improves the security of the IIoT environment. Through comprehensive experimentation on both UNSW-NB15 and UCI SECOM datasets, the model exhibited exceptional performance, achieving an average accuracy of 99.55% and 98.87%, precision of 98.90% and 98.93%, recall of 98.87% and 98.80%, and F-score of 98.88% and 98.87% for the respective datasets. The IID-SBODL model contributes to the development of robust intrusion detection mechanisms for safeguarding critical industrial processes in the era of interconnected and smart IIoT environments.

Research on Credit Risk Prediction Method of Blockchain Applied to Supply Chain Finance

INTRODUCTION: From the perspective of blockchain, it establishes a credit risk evaluation index system for supply chain finance applicable to blockchain, constructs an accurate credit risk prediction model, and provides a reliable guarantee for the research of credit risk in supply chain finance.OBJECTIVES: To address the inefficiency of the current credit risk prediction and evaluation model for supply chain finance.METHODS: This paper proposes a combined blockchain supply chain financial credit risk prediction and evaluation method based on kernel principal component analysis and intelligent optimisation algorithm to improve Deep Echo State Network. Firstly, the evaluation system is constructed by describing the supply chain financial credit risk prediction and evaluation problem based on blockchain technology, analysing the evaluation indexes, and constructing the evaluation system; then, the parameters of DeepESN network are optimized by combining the kernel principal component analysis method with the JSO algorithm to construct the credit risk prediction and evaluation model of supply chain finance; finally, the effectiveness, robustness, and real-time performance of the proposed method are verified by simulation experiment analysis.RESULTS: The results show that the proposed method improves the prediction efficiency of the prediction model.CONCLUSION: The problems of insufficient scientific construction of index system and poor efficiency of risk prediction model of B2B E-commerce transaction size prediction method are effectively solved.

PAR-Net: An Enhanced Dual-Stream CNN-ESN Architecture for Human Physical Activity Recognition.

Physical exercise affects many facets of life, including mental health, social interaction, physical fitness, and illness prevention, among many others. Therefore, several AI-driven techniques have been developed in the literature to recognize human physical activities. However, these techniques fail to adequately learn the temporal and spatial features of the data patterns. Additionally, these techniques are unable to fully comprehend complex activity patterns over different periods, emphasizing the need for enhanced architectures to further increase accuracy by learning spatiotemporal dependencies in the data individually. Therefore, in this work, we develop an attention-enhanced dual-stream network (PAR-Net) for physical activity recognition with the ability to extract both spatial and temporal features simultaneously. The PAR-Net integrates convolutional neural networks (CNNs) and echo state networks (ESNs), followed by a self-attention mechanism for optimal feature selection. The dual-stream feature extraction mechanism enables the PAR-Net to learn spatiotemporal dependencies from actual data. Furthermore, the incorporation of a self-attention mechanism makes a substantial contribution by facilitating targeted attention on significant features, hence enhancing the identification of nuanced activity patterns. The PAR-Net was evaluated on two benchmark physical activity recognition datasets and achieved higher performance by surpassing the baselines comparatively. Additionally, a thorough ablation study was conducted to determine the best optimal model for human physical activity recognition.

Chaotic Dynamical Enhanced Optical Physical Layer Encryption in OFDM-PON System Based on Echo State Network

Chaotic Dynamical Enhanced Optical Physical Layer Encryption in OFDM-PON System Based on Echo State Network

The connectivity degree controls the difficulty in reservoir design of random boolean networks.

Reservoir Computing (RC) is a paradigm in artificial intelligence where a recurrent neural network (RNN) is used to process temporal data, leveraging the inherent dynamical properties of the reservoir to perform complex computations. In the realm of RC, the excitatory-inhibitory balance b has been shown to be pivotal for driving the dynamics and performance of Echo State Networks (ESN) and, more recently, Random Boolean Network (RBN). However, the relationship between b and other parameters of the network is still poorly understood. This article explores how the interplay of the balance b, the connectivity degree K (i.e., the number of synapses per neuron) and the size of the network (i.e., the number of neurons N) influences the dynamics and performance (memory and prediction) of an RBN reservoir. Our findings reveal that K and b are strongly tied in optimal reservoirs. Reservoirs with high K have two optimal balances, one for globally inhibitory networks (b < 0), and the other one for excitatory networks (b > 0). Both show asymmetric performances about a zero balance. In contrast, for moderate K, the optimal value being K = 4, best reservoirs are obtained when excitation and inhibition almost, but not exactly, balance each other. For almost all K, the influence of the size is such that increasing N leads to better performance, even with very large values of N. Our investigation provides clear directions to generate optimal reservoirs or reservoirs with constraints on size or connectivity.

Comparative Analysis of Linear Models and Artificial Neural Networks for Sugar Price Prediction

Sugar is an important commodity that is used beyond the food industry. It can be produced from sugarcane and sugar beet, depending on the region. Prices worldwide differ due to high volatility, making it difficult to estimate their forecast. Thus, the present work aims to predict the prices of kilograms of sugar from four databases: the European Union, the United States, Brazil, and the world. To achieve this, linear methods from the Box and Jenkins family were employed, together with classic and new approaches of artificial neural networks: the feedforward Multilayer Perceptron and extreme learning machines, and the recurrent proposals Elman Network, Jordan Network, and Echo State Networks considering two reservoir designs. As performance metrics, the MAE and MSE were addressed. The results indicated that the neural models were more accurate than linear ones. In addition, the MLP and the Elman networks stood out as the winners.

Stocks price prediction based on optimized echo state network by sparrow search algorithm

Stocks price prediction based on optimized echo state network by sparrow search algorithm

Brain-inspired modular echo state network for EEG-based emotion recognition.

Previous studies have successfully applied a lightweight recurrent neural network (RNN) called Echo State Network (ESN) for EEG-based emotion recognition. These studies use intrinsic plasticity (IP) and synaptic plasticity (SP) to tune the hidden reservoir layer of ESN, yet they require extra training procedures and are often computationally complex. Recent neuroscientific research reveals that the brain is modular, consisting of internally dense and externally sparse subnetworks. Furthermore, it has been proved that this modular topology facilitates information processing efficiency in both biological and artificial neural networks (ANNs). Motivated by these findings, we propose Modular Echo State Network (M-ESN), where the hidden layer of ESN is directly initialized to a more efficient modular structure. In this paper, we first describe our novel implementation method, which enables us to find the optimal module numbers, local and global connectivity. Then, the M-ESN is benchmarked on the DEAP dataset. Lastly, we explain why network modularity improves model performance. We demonstrate that modular organization leads to a more diverse distribution of node degrees, which increases network heterogeneity and subsequently improves classification accuracy. On the emotion arousal, valence, and stress/calm classification tasks, our M-ESN outperforms regular ESN by 5.44, 5.90, and 5.42%, respectively, while this difference when comparing with adaptation rules tuned ESNs are 0.77, 5.49, and 0.95%. Notably, our results are obtained using M-ESN with a much smaller reservoir size and simpler training process.

Adaptive fuzzy echo state network optimal synchronization control of hybrid–order chaotic systems via reinforcement learning

In this paper, a novel optimal synchronization control scheme for fractional–integer hybrid–order chaotic systems is formulated. To deal with the fractional–order (FO) constraint, a transformation programme is developed and then the master system considered as an FO chaotic system is transformed into an integer–order one. A fuzzy echo state network (FESN) with the advantages of both fuzzy logic system and echo state network is introduced to approximate system uncertainty. Simultaneously, to alleviate the resource pressure, an optimal synchronization control is proposed in the light of the reinforcement learning mechanism, in which critic–actor update laws are constructed through the negative gradient of a positive function with regard to Bellman residual. Compared to typical fuzzy synchronization control, the devised FESN synchronization scheme with the same computational complexity possess superior approximation ability and synchronization performance. Ultimately, three simulation cases are exhibited to check the validity of the proposed approach.